fix(compression): fit summarizer input to small aux windows instead of lowering the session threshold - #84325
Conversation
…f lowering the session threshold A summary model whose context window is smaller than the session threshold used to trigger an auto-lower of the live threshold to the aux window — halving a 272K main model's usable context when paired with a 128K compression model: compaction fires at ~47% of the real window, twice as often, discarding conversation detail early and breaking the prompt-cache prefix each time. The comparison itself was against the wrong quantity: the summariser request is already bounded (per-message truncation plus the _SUMMARY_INPUT_MAX_CHARS aggregate cap), so it never approaches the session threshold. What actually needs to fit the aux window is that bounded request — not the conversation. check_compression_model_feasibility now stashes the resolved aux window on the compressor and leaves the threshold alone; the 64K hard floor is unchanged. _generate_summary assembles the prompt through a single builder and, when the stashed window is smaller than the assembled request plus the summary output budget, shrinks the serialized-turns block (head+tail, explicit omitted-middle marker — the same shape as the existing aggregate cap) until it fits. estimate_tokens_rough over-counts, so a passing fit cannot overflow the real window; a still-oversized pathological prompt is sent best-effort and a failure routes through the existing main-model fallback. _bound_summary_input gains an optional max_chars override used by the fit loop.
alt-glitch
added
type/bug
PR: fix(compression): fit summarizer input to small aux windows instead of lowering the session threshold
|
Addresses automated-review findings on the window-fit change: * Log when the trim loop exits via its iteration cap while the prompt is still over the fit budget — previously only the content-floor exit warned, and a non-converged prompt was sent silently. (Reaching the cap without converging requires a scaffold-dominated prompt, so this is defensive, but a silently oversized send deserves a trace.) * Restore a user-facing diagnostic for small aux windows, without the old auto-lower warning's noise: when the per-call fit actually trims the summarizer input, compress_context emits a one-time-per-session status pointing at auxiliary.compression.model — mirroring the existing _last_summary_fallback_used surfacing pattern. Sessions whose windows never need trimming see nothing.
|
Thanks for the review — all four points addressed or answered:
|
This branch has not been deployed
3 participants
What does this PR do?
Stops a small-context auxiliary compression model from silently halving the main model's usable context window.
Since #12898, when
auxiliary.compression.modelresolves to a window smaller than the session's compression threshold, the feasibility check auto-lowers the live session threshold to the aux window. Pairing a 272K main model with a 128K compression model (a natural choice — small models are fast and cheap summarizers) drops the compaction trigger from 231K to 128K: compaction fires at ~47% of the real window, roughly twice as often, each pass discarding conversation detail and invalidating the prompt-cache prefix. Measured on a live gateway, enabling a 128K summary model took a session from ~1 compaction/day to 20+/day with no visible cause beyond a one-line startup log.The comparison itself is against the wrong quantity. The summarizer request is already bounded — per-message truncation (
_CONTENT_MAX6K chars/message) plus the_SUMMARY_INPUT_MAX_CHARS(160K chars) aggregate cap with an explicit omitted-middle marker — so it never approaches the session threshold it is compared to. What must fit the aux window is that bounded request plus the summary output budget, not the conversation.The change:
check_compression_model_feasibilityno longer touches the threshold (ortail_token_budget/threshold_percent). It stashes the resolved aux window on the compressor (summary_model_context_length) and logs one INFO line. The 64K hard floor and the no-provider warning are unchanged._generate_summaryassembles the prompt through a single builder and, when the stashed window is smaller than the assembled request plus the output budget, shrinks the serialized-turns block — head + tail with the existing omitted-middle marker shape — until the whole prompt fits (estimate_tokens_roughover-counts every content class, so a passing fit cannot overflow the real window). A pathological prompt that cannot be shrunk below the floor is sent best-effort; a failure routes through the existing main-model fallback (_fallback_to_main_for_compression), exactly as other aux failures do._bound_summary_inputgains an optionalmax_charsoverride used by the fit loop.Net effect: a small aux model now summarizes a per-call-bounded digest while the main conversation keeps its full window. The threshold — since #80997/#81069 compared against projected real usage — stays a property of the main model only.
Related: #12898 (introduced the auto-lower alongside the 64K floor — the floor is kept), #67422 (the threshold-suggestion math in the removed warning; the suggestion is no longer needed because nothing needs correcting), #8499 (
auxiliary.compression.context_lengthconfig override — still honored, it feeds the stashed window), #52392 (fallback-chain context screening — untouched).Related Issue
No open issue describes this defect; the origin PR and adjacent reports are linked above.
Type of Change
Changes Made
agent/conversation_compression.py— feasibility check stashes the aux window instead of lowering the session threshold; removes the auto-lower bookkeeping (tail_token_budgetlockstep,threshold_percentrewrite) and its multi-line user warning, which no longer has anything to warn about.agent/context_compressor.py— newsummary_model_context_lengthfield;_bound_summary_input(max_chars=...)override;_generate_summaryprompt assembly extracted into one builder + per-call fit loop with a floor guard and best-effort logging.tests/agent/test_summary_window_fit.py— new: feasibility stash semantics (threshold untouched, no warning), per-call trim fits the window on CJK-dense content, fit inactive without a stashed window or when the main model summarizes,max_charsoverride.tests/run_agent/test_compression_feasibility.py— three tests updated from auto-lower semantics to stash-and-fit semantics; the gateway-replay test now exercises the surviving no-provider warning; hard-floor rejection and config-override tests unchanged.How to Test
pytest tests/agent/test_summary_window_fit.py tests/run_agent/test_compression_feasibility.py -q— 16 passed.pytest tests/agent tests/run_agent -q -k "compress or compaction or summary or feasib"— 711 passed on this branch.auxiliary.compression.{provider,model}to any 128K model under a 200K+ main model. Before: startup logsauto-lowered session threshold to 128000and long sessions compact at ~half the real window, several times per hour. After: the threshold log line is gone,Preflight compressionstill fires at the configured threshold, and when compaction runs the log showsSummarizer input trimmed to fit <model>'s 128000-token windowwhile the summary call succeeds within it.Checklist
Code
fix(scope):,feat(scope):, etc.)tests/run on my machine has environment-dependent failures (i18n catalogs, models.dev fetch, etc.) that are identical on cleanmainDocumentation & Housekeeping
_bound_summary_input, the fit loop, and the feasibility stashcli-config.yaml.exampleif I added/changed config keys — N/A (no config changes; existing keys keep their meaning)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/A